Description

Resolves #1816 by implementing configurable limits for maximum redirects and maximum response size in the crawler backend.

Changes Made

• Added max_redirects (default 10) and max_size (default 5MB) parameters to the crawl_target function.
• Switched request model from client.get to client.stream("GET", url) for chunked stream validation.
• Implemented proactive size checks on the Content-Length header (if present).
• Implemented cumulative download size validation across stream chunks to terminate responses exceeding max_size.
• Added a dedicated unit test suite in testing/backend/unit/test_crawler_limits.py testing redirect limits, content-length limits, and streaming chunk boundaries. All 88 tests pass successfully.